Image processing method and image processing apparatus

ABSTRACT

There is provided an image processing method of laying out and outputting an image of an original formed from a plurality of pages on one page, the method comprising: inputting image data of the original formed from the plurality of pages; dividing the image data input in the inputting into a plurality of blocks; discarding unnecessary information; appending necessary information to each block obtained by division in the dividing based on the predetermined rule; replacing the respective blocks obtained by division in the dividing with the block from which the unnecessary information is discarded in the discarding, and the block to which the necessary information is appended in the appending; and outputting the respective blocks which have undergone replacement in the replacing by laying out the respective blocks on one page.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing method and imageprocessing apparatus for editing and outputting image data acquired froman input device.

2. Description of the Related Art

As low-cost scanner devices become available, an original can be easilyscanned and converted into digital data even at home. Inkjet printers asprinting apparatuses have been developed, and even a multifunctionalperipheral (MFP) called a multi-function printer capable of not onlyprinting an image, but also scanning an image becomes popular. The usercan use a scanner device and printer device to easily copy his originalsand contents at home.

An example of contents which improve user's convenience when copied is“sheet music”. In general, the user purchases sheet music as a book.However, it impairs user's convenience to set sheet music purchased as abook directly on a music stand. For example, when the user sets sheetmusic of many pages directly on a music stand to see a desired page, hemust forcibly open the sheet music, damaging the sheet music. To preventthis, a music stand devised to fix sheet music at a desired page is alsocommercially available. However, such a music stand makes it difficultto turn a page. It is generally cumbersome to turn a page of sheet musicduring the performance. This work is very troublesome especially duringthe performance of a musical instrument using both hands, such as thepiano.

During the music lesson of the piano or the like, the teacher oftenwrites an instruction directly on sheet music. It is popular to copysheet music and keep the original clean.

When editing and distributing sheet music of a composition by a user orwhen editing music composed in response to user's request anddistributing it as sheet music, various editing operations are required.For this purpose, a technique for, for example, editing sheet music isdisclosed.

To reduce the number of page turns, a musician sometimes arranges andprints a plurality of pages of sheet music on one paper sheet. By usingthe layout copy function of an MFP, a plurality of pages of sheet musicas a purchased book can be arranged and copied on one paper sheet. Whensheet music data is saved in a PC, the player can designate layoutprinting via a printer driver to print a plurality of pages of sheetmusic on one paper sheet.

The following techniques have been disclosed as patents regarding layoutprinting and editing of an image such as sheet music.

Japanese Patent Laid-Open No. 2002-144664 discloses a method of dividinga 1-page original image into a predetermined number of output areas,arranging different types of documents in the respective output areas incorrespondence with each other, and outputting the documents.

Japanese Patent No. 2879941 discloses a method of resizing a given arearepresented by a five-line staff of sheet music and laying it out sothat the resized area represented by a five-line staff width andfive-line staff length becomes homothetic to the given area.

Japanese Patent No. 3169142 discloses a method of designating theboundary between parts of sheet music or that between bars, moving theposition of the boundary so that the designated boundary coincides withthe screen display, and displaying the sheet music.

Japanese Patent Laid-Open Nos. 06-149235 and 06-149236 disclose a methodof aligning in the lateral direction the left ends of the five-linestaffs of simultaneously played parts in accordance with the coordinatesof the top and bottom lines of the five-line staffs of the parts.

Sheet music as a general book describes information for a performancebased on rules built up through historical development. Sheet music isnot convenient for all players. Therefore, a player edits sheet music soas to easily play for himself by, for example, copying sheet music anddividing it into a plurality of paper sheets so as to allow easy pageturning, by coloring sheet music, or by writing signs on it. The editingwork optimizes sheet music for the player. However, it is verycumbersome for the player to edit sheet music, so demands have arisenfor easy editing of sheet music.

FIG. 24 shows a layout when arranging and printing data from four pageson one paper sheet. The user can designate layouts 1-a to 1-h via a UI(User Interface). In FIG. 24, reference numeral 1001 denotes an output;1002, an image corresponding to data of the first page; 1003, an imagecorresponding to data of the second page; 1004, an image correspondingto data of the third page; and 1005, an image corresponding to data ofthe fourth page. The layout 1-a shows that the image 1002 is arranged atthe upper left on the output 1001, the image 1003 is arranged at thelower left, the image 1004 is arranged at the upper right, and the image1005 is arranged at the lower right.

However, this layout printing simply rearranges images corresponding torespective pages for each page. For example, when the output 1001 issheet music, visibility sometimes becomes poor.

SUMMARY OF THE INVENTION

The present invention has been made to overcome the conventionaldrawbacks, and has as its feature to provide an image processing methodand image processing apparatus capable of changing the arrangement ofthe original images of a plurality of pages not for each page but withhigh degree of freedom and laying them out on one page by simple editingwork.

According to an aspect of the present invention, there is provided animage processing method of laying out and outputting an image of anoriginal formed from a plurality of pages on one page. The methodincludes inputting image data of the original formed from the pluralityof pages; dividing the image data input in the inputting into aplurality of blocks; selecting unnecessary information from each blockobtained by division in the dividing based on a predetermined rule tomaintain continuity of the original formed from the plurality of pageswhen laying out, on one page, the image data of the original formed fromthe plurality of pages; discarding the unnecessary information selectedin the selecting; appending necessary information to each block obtainedby division in the dividing based on the predetermined rule; replacingthe respective blocks obtained by division in the dividing with theblock from which the unnecessary information is discarded in thediscarding, and the block to which the necessary information is appendedin the appending; and outputting the respective blocks which haveundergone replacement in the replacing by laying out the respectiveblocks on one page.

According to another aspect of the present invention, there is providedan image processing apparatus which lays out and outputs an image of anoriginal formed from a plurality of pages on one page. The apparatusincludes an input unit which inputs image data of the original formedfrom the plurality of pages; a division unit which divides the imagedata input by the input unit into a plurality of blocks; a selectionunit which selects unnecessary information from each block obtained bydivision by the division unit based on a predetermined rule to maintaincontinuity of the original formed from the plurality of pages whenlaying out, on one page, the image data of the original formed from theplurality of pages; a discard unit which discards the unnecessaryinformation selected by the selection unit; an appending unit whichappends necessary information to each block obtained by division by thedivision unit based on the predetermined rule; a replacement unit whichreplaces the respective blocks obtained by division by the division unitwith the block from which the unnecessary information is discarded bythe discard unit, and the block to which the necessary information isappended by the appending unit; and an output unit which outputs therespective blocks which have undergone replacement by the replacementunit by laying out the respective blocks on one page.

According to yet another aspect of the present invention, there isprovided a computer-readable medium containing computer-executableinstructions utilized in an image processing apparatus configured to layout and output an image of an original formed from a plurality of pageson one page. Here, the computer-readable medium includescomputer-executable instructions for inputting image data of theoriginal formed from the plurality of pages; computer-executableinstructions for dividing the image data input in the inputting into aplurality of blocks; computer-executable instructions for selectingunnecessary information from each block obtained by division in thedividing based on a predetermined rule to maintain continuity of theoriginal formed from the plurality of pages when laying out, on onepage, the image data of the original formed from the plurality of pages;computer-executable instructions for discarding the unnecessaryinformation selected in the selecting; computer-executable instructionsfor appending necessary information to each block obtained by divisionin the dividing based on the predetermined rule; computer-executableinstructions for replacing the respective blocks obtained by division inthe dividing with the block from which the unnecessary information isdiscarded in the discarding, and the block to which the necessaryinformation is appended in the appending; and computer-executableinstructions for outputting the respective blocks which have undergonereplacement in the replacing by laying out the respective blocks on onepage.

According to the present invention, when laying out and printing theimages of a plurality of pages on one paper sheet, not only the imagescan be simply rearranged for each page, but also the layout can bechanged with high degree of freedom to improve user visibility, therebyprinting the images.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual view showing an example of the configuration ofan image processing system including an image processing apparatusaccording to the present invention;

FIG. 2 is a block diagram of the image processing apparatus in FIG. 1;

FIG. 3 is a schematic perspective view of the outer appearance of an MFPaccording to an embodiment of the present invention;

FIG. 4 is a block diagram of the MFP in FIG. 3;

FIG. 5 is a flowchart of image processing executed in copying;

FIG. 6 is a flowchart showing the operation sequence of an imageprocessing method according to the present invention;

FIG. 7 is a view for explaining skew correction of an original image;

FIG. 8 is a view schematically showing the original image of sheetmusic;

FIG. 9 is a view schematically showing the result of dividing theoriginal image of sheet music into blocks;

FIG. 10 is a view schematically showing the result of numbering blocks;

FIG. 11 is a view showing an example of discarding unnecessaryinformation;

FIG. 12 is a view showing an example of appending necessary information;

FIG. 13 is a view schematically showing the result of executing layoutprinting according to the present invention;

FIG. 14 is a flowchart showing an operation sequence in block numberingprocessing;

FIG. 15 is a view showing the original image of one page;

FIG. 16 is a view schematically showing numbered divided blocks;

FIG. 17 is a view schematically showing an example of grouping blocks;

FIG. 18 is a view for explaining group division processing;

FIG. 19 is a view showing an original image in which blocks of the textpart in the original image are divided;

FIG. 20 is a flowchart showing a processing sequence to display a layoutresult and determine a layout;

FIG. 21 is a view schematically showing the columns of a text;

FIG. 22 is a view schematically showing a layout result for performinglayout printing;

FIG. 23 is a view schematically showing a layout result for performinglayout printing;

FIG. 24 is a view showing layouts for layout printing; and

FIG. 25 is a view schematically showing a layout result for performinglayout printing.

DESCRIPTION OF THE EMBODIMENTS

Various embodiments of the present invention will be described in detailbelow with reference to the accompanying drawings.

<PC Application>

An embodiment for carrying out the present invention by applicationsoftware running on a personal computer (to be referred to as a PChereinafter) serving as a host apparatus will be explained.

FIG. 1 is a conceptual view showing an example of the configuration ofan image processing system including an image processing apparatusaccording to the present invention.

An image processing system 101 in FIG. 1 includes a scanner 102 servingas an image reading apparatus, a PC 103, and a printer 104 serving as aprinting apparatus. The PC 103 can receive/output image data via anetwork 105. It is possible to change the layout of sheet music scannedby the scanner 102 and print the resultant sheet music by the printer104.

FIG. 2 is a block diagram of the PC 103 shown in FIG. 1.

In FIG. 2, reference numeral 201 denotes a ROM (Read Only Memory); 202,a CPU (Central Processing Unit); and 203, a RAM (Random Access Memory).Reference numeral 204 denotes an I/O (Input/Output) unit; 205, a NIC(Network Interface Card); 206, an HDD (Hard Disk Drive); and 207, a bus.

The CPU 202 controls the operation of the PC 103 via the bus 207 inaccordance with a program and data stored in the ROM 201. At the sametime, the CPU 202 performs image processing using the RAM 203 as a workmemory. For example, the CPU 202 performs image processing for imagedata input via the I/O unit 204 or NIC 205, or image data stored inadvance in a storage medium such as the HDD 206. The CPU 202 outputs theprocessed image data via the I/O unit 204 or NIC 205, or stores it in astorage medium such as the HDD 206.

For example, the I/O unit 204 is connected via a predetermined interfaceto an image input/output device such as a monitor (e.g., CRT or LCD), aprinter, or a scanner, or a storage device having a storage medium suchas a magnetic disk or optical disk. The I/O unit 204 can receive/outputimage data via the NIC 205 from/to the above-mentioned imageinput/output device and a computer device connected to the storagedevice. Examples of the network are Ethernet, FDDI (Fiber DistributedData Interface), IEEE1394 serial bus, and USB (Universal Serial Bus).

<MFP>

The same effects as the above-described ones can also be obtained byusing an image processing system in which a single multi-functionprinter (to be referred to as an MFP hereinafter) operates without themediacy of a PC.

FIG. 3 is a schematic perspective view of the outer appearance of an MFP301 serving as a multifunctional peripheral according to an embodimentof the present invention. The MFP 301 has the function of a general PCprinter which receives data from a PC and prints it, and the function ofa scanner. Further, the MFP 301 has, as a function executed by a singleMFP, a copy function of printing, by a printer, an image scanned by ascanner, a function of directly reading image data stored in a memorycard or the like and printing it, or a function of receiving image datafrom a digital camera and printing it.

In FIG. 3, the MFP 301 includes a reading unit 302 such as a flat-bedscanner, a printing unit 303 of an inkjet type, electro-photography, orthe like, a display unit 304 serving as a display panel, and anoperation unit 305 serving as an operation panel having various keys,switches, and the like. A USB port (not shown) is provided on the rearsurface of the MFP 301 as a PC interface for communicating with a PC,and enables communication with a PC. In addition to these components,the MFP 301 includes a card interface 306 serving as a card slot forreading out data from a variety of memory cards, and a camera interface307 serving as a camera port for communicating data with a digitalcamera. Further, the MFP 301 includes an auto document feeder (to bereferred to as an ADF hereinafter) 308 for automatically setting anoriginal onto the original table.

FIG. 4 is a block diagram of the MFP 301 shown in FIG. 3.

In FIG. 4, a CPU 401 controls various functions of the MFP 301, andexecutes an image processing program stored in a ROM 406 in accordancewith a predetermined operation to the operation unit 305.

The reading unit 302 having a CCD reads an original image to outputanalog luminance data of red (R), green (G), and blue (B). The readingunit 302 may also include a contact image sensor (CIS) instead of theCCD. When the ADF 308 as shown in FIG. 3 is provided, the reading unit302 can successively read the original images of order sheets set on theADF 308.

The card interface 306 receives, in accordance with a predeterminedoperation to the operation unit 305, image data captured by a digitalcamera or the like and recorded on a memory card or the like. Ifnecessary, an image processing unit 402 converts image data received viathe card interface 306. For example, the image processing unit 402converts image data corresponding to the color space (e.g., YCbCr) of adigital camera into one corresponding to the RGB color space (e.g.,NTSC-RGB or sRGB). If necessary, the received image data undergoesvarious processes necessary for an application, such as resolutionconversion into an effective pixel count based on header information ofthe image data.

A camera interface 307 is directly connected to a digital camera to readimage data.

The image processing unit 402 performs image processes (to be describedlater) such as image analysis, calculation of the conversioncharacteristic, conversion from a luminance signal (RGB) into a densitysignal (CMYK), scaling, gamma conversion, and error diffusion. Dataobtained by these image processes are stored in a RAM 407. Whencorrected data stored in the RAM 407 reach a predetermined amountnecessary to print by the printing unit 303, the printing unit 303executes a print operation.

A nonvolatile RAM 408 is, for example, a battery backed-up SRAM, andstores data unique to the MFP 301 and the like.

The operation unit 305 has a photo direct print start key to selectimage data stored in a storage medium and start printing, a key to printan order sheet, and a key to read an order sheet. The operation unit 305also has a copy start key to perform monochrome copying or colorcopying, a mode key to designate a mode such as resolution or imagequality in copying, a stop key to stop a copy operation or the like, anumeric keypad to input the number of copies, a registration key, andthe like. The CPU 401 detects the pressed states of these keys, andcontrols each unit in accordance with the states.

The display unit 304 includes a dot matrix type liquid crystal display(LCD) and LCD driver, and presents various displays under the control ofthe CPU 401. The display unit 304 displays the thumbnail of image datastored in a storage medium.

The printing unit 303 includes an inkjet head of an inkjet type,general-purpose IC, and the like. The printing unit 303 reads out printdata stored in the RAM 407, and prints it out as a hard copy under thecontrol of the CPU 401.

A driving unit 411 includes a stepping motor for driving feed anddelivery rollers in the operations of the reading unit 302 and printingunit 303, a gear for transmitting the driving force of the steppingmotor, and a driver circuit for controlling the stepping motor.

A sensor unit 410 includes a sensor for detecting the width of a printmedium, a sensor for detecting the presence/absence of a print medium, asensor for detecting the width of an original image, a sensor fordetecting the presence/absence of an original image, and a sensor fordetecting the type of print medium. Based on pieces of informationobtained from these sensors, the CPU 401 detects the states of anoriginal and print medium.

A PC interface 414 is an interface between the PC and the MFP 301. TheMFP 301 executes operations such as printing and scanning based oninstructions transmitted from the PC via the PC interface 414.

In copying, image data of an original image read by the reading unit 302is processed in the MFP 301, and printed by the printing unit 303. Whenthe user designates a copy operation via the operation unit 305, thereading unit 302 reads an original set on the original table. Image dataof the read original image is transmitted to the image processing unit402, and undergoes image processing (to be described later). Theprocessed image data is transmitted to the printing unit 303, andprinted.

FIG. 5 is a flowchart of image processing executed in copying.Respective steps will be explained, but a detailed description of stepswhich are not essential to the present invention will be omitted.

In step S110, image data which is read by the reading unit 302 andA/D-converted undergoes shading correction to correct variations ofimage sensors.

In step S120, the image data undergoes input device color conversion toconvert image data corresponding to a color space unique to the readingunit 302 serving as an input device into image data corresponding to astandard color space. More specifically, the image data is convertedinto image data corresponding to a color space such as sRGB defined byIEC (International Electrotechnical Commission) or AdobeRGB proposed byAdobe Systems. The conversion method is, for example, an arithmeticmethod based on a 3×3 or 3×9 matrix, or a look-up table method oflooking up to a table describing a conversion rule and determining aresultant value based on the table.

In step S130, the image data having undergone color conversion undergoesimage correction/manipulation processing. The processing contentsinclude edge emphasis processing of correcting blurring generated uponreading an original image, processing of removing offset generated uponreading by light irradiation, and character manipulation processing forimproving character readability.

In step S140, enlargement/reduction processing is executed to convertthe image data at a desired ratio when the user designates resizing orin layout copying of laying out two original sheets on one paper sheet.The conversion method is generally a bicubic method, nearest neighbormethod, or the like. When laying out and printing a plurality of imageson one print medium in layout copying or the like, the operations insteps S110 to S140 are repeated to read a plurality of images and layout the read images on one page. Then, the process shifts to thefollowing print operation.

In step S150, the image data corresponding to the standard color spaceis converted into image data corresponding to a color space unique to anoutput device. Similar to step S120, the conversion method suffices tobe the arithmetic method based on a matrix or the look-up table method.Image data corresponding to the color space unique to the output deviceis converted into one corresponding to the color space of the colors ofinks used in an MFP 301 of inkjet type, such as cyan, magenta, yellow,and black.

In step S160, the image data undergoes quantization processing. Forexample, when an ink dot is expressed by a binary value representingwhether or not to discharge an ink dot, the image data suffices to bebinarized according to a quantization method such as error diffusion. Asa result, the image data is converted into a data format printable bythe printing unit. The print operation is executed based on the imagedata of this data format, forming an image.

First Embodiment

A first embodiment of the present invention will be described belowreferring to the drawings. FIG. 6 is a flowchart showing the operationsequence of an image processing method according to the presentinvention.

First, in step S210, the number N of pages of original images that isdesignated for layout by the user using an operation unit 305 of an MFP301 or the like is substituted as a Count value.

Then, in step S220, image data of one page of an original imageundergoes pre-processing. As the pre-processing, skew correction of anoriginal image is executed first. This is because the reading device mayread an original image with a skew. When the original image is sheetmusic, skew correction suffices to use a method of correcting a skewusing a five-line staff of the sheet music. For example, the imagedensity of read sheet music is calculated at a plurality of angles tocreate a histogram, as shown in FIG. 7. When the angle is parallel tothe staff, the histogram draws steep curves at portions corresponding tothe five lines of the five-line staff. From this, an angle at which thefive-line staff becomes parallel to the reading direction is calculated,and the entire image is rotated and corrected. If necessary, abinarization operation may also be done using a threshold. Although afixed threshold may also be used, the threshold may also be calculatedafter creating a brightness histogram when the highest image density ofan original image changes between pages or the image density of a marginon an original changes between pages. Filtering processing may also bedone to thicken the thin lines of a five-line staff and hook, improvingtheir reproducibility. The reference position of the original image isdetermined using five-line staff information in the skew-corrected imagedata.

In step S230, the image data of the original image is divided into aplurality of rectangular blocks. In this step, image area separation isexecuted to separate the image into at least three types of attributes“text part”, “natural image part”, and “background part”. Each separatedimage is divided into rectangular blocks for each attribute. Forexample, when the original image expressed by the image data havingundergone pre-processing in step S220 is an original image in FIG. 8,the original image divided into rectangular blocks for respectiveattributes “text part” and “background part” by image area separation isas shown in FIG. 9. Since the original image is sheet music, it isdivided into blocks for respective bars. The bar line of a bar is astraight line perpendicular to the five-line staff. Such straight linesare detected, and the original image is divided into areas defined bythe detected straight lines. As a result, the original image can bedivided into blocks for respective bars.

In step S240, the divided rectangular blocks are numbered based on arule made in advance to maintain the original continuity of a pluralityof pages. In this step, the respective rectangular blocks of a 1-pageoriginal image are numbered in ascending order from an upper left blocktoward a lower right block. FIG. 10 shows the result of numbering on theoriginal image divided into rectangular blocks in FIG. 9. Therectangular blocks are numbered sequentially from the upper left. Ablock 901 is numbered with “1-1”, a block 902 is numbered with “1-2”, ablock 903 is numbered with “1-3”, a block 904 is numbered with “1-4”, ablock 905 is numbered with “1-5”, and a block 906 is numbered with“1-33”. A block 907 is numbered with “1-10”, a block 908 is numberedwith “1-16”, a block 909 is numbered with “1-22”, a block 910 isnumbered with “1-28”, a block 911 is numbered with “1-13”, and a block912 is numbered with “1-31”. When the original image is sheet music, itis divided into blocks for the page number, title, composer, greatstaffs, bars, and the like. The coordinates of each block in theoriginal image of one page is obtained, and the block is numbered inaccordance with the coordinates.

Next, in step S250, information which becomes unnecessary in layout isselected based on the rule made in advance to maintain the originalcontinuity of a plurality of pages and, block-level information as wellas some information contained within a block are discarded. For example,when the original images of a plurality of pages are divided into blocksand blocks are laid out on one page according to the present invention,the page number of an original image is unnecessary information. Thus,information of the block 901 in FIG. 10 is discarded. Sheet music isexpressed by great staffs, each of which combines treble and bass staffsusing a brace. At the beginning of a great staff, a brace, G clef, and Fclef are described. A case where the original images of two pages arelaid out and printed on one page will be exemplified. The sheet music ofFIG. 8 as the original image of the first page includes five greatstaffs each made up of six bars. The original image of the second pageincludes a page number, and seven great staffs each made up of six bars.The second page is divided into block “2-1” of a page number, and blocks“2-2” to “2-43” of respective bars. In layout according to the firstembodiment, as represented by a schematic layout shown in FIG. 13, eachgreat staff includes nine bars. Before the layout change, the blocks907, 908, and 910 are positioned at the beginnings of respective greatstaffs, as shown in FIG. 10. After the layout change, these blocks movefrom the beginnings of great staffs, as shown in FIG. 13. For thisreason, braces, G clefs, and F clefs in these blocks become unnecessary.FIG. 11 shows an example of an image from which a brace, G clef, and Fclef which become unnecessary in a block are discarded. A bar 1101represents a bar at the beginning of a great staff. A bar 1102represents a bar from which a brace, G clef, and F clef which becomeunnecessary in a block are discarded. A brace, G clef, and F clef arepositioned at the beginning of a great staff, and thus always containedin a block at the left end. A brace, G clef, and F clef can be detectedby executing pattern matching in a block.

In step S260, information which becomes necessary in layout is appendedbased on the rule made in advance to maintain the original continuity ofa plurality of pages. As described above, a brace, G clef, and F clefare necessary at the beginning of a great staff. After the layoutchange, the blocks 911 and 912 move to the beginnings of great staffs,and require braces, G clefs, and F clefs. FIG. 12 shows an example of animage to which a brace, G clef, and F clef which become necessary in ablock are appended. A bar 1201 represents a bar which does not appear atthe beginning of a great staff. A bar 1202 represents a bar to which abrace, G clef, and F clef which become necessary in a block areappended. Necessary information suffices to be appended in accordancewith sheet music after layout change by storing image data of necessaryinformation such as a brace, G clef, and F clef in advance in the memoryof the MFP 301.

The Count value is decremented in step S270, and it is determined instep S280 whether the Count value has reached 0. If it is determinedthat the Count value has not reached 0 (YES in step S280), steps S220 toS270 are repeated for the original image of the next page. If itdetermined that the Count value has reached 0 (NO in step S280), theprocess advances to step S290.

In step S290, the layout of numbered rectangular blocks is changed inaccordance with a user instruction regarding the number of blocks to belaid out. As described above, FIG. 13 is a view schematically showing alayout when laying out and printing the original images of two pages onone page. Blocks numbered with “1-2” to “1-33” are laid out from theoriginal image of the first page. Blocks from a block 913 numbered with“2-2” to a block 914 numbered with “2-43” are laid out from the originalimage of the second page (not shown). These blocks are those replacedwith blocks to and from which information is discarded and appended insteps S250 and S260.

In the first embodiment, the layout of sheet music of two pagesincluding 6-bar great staffs is changed into sheet music of one pageincluding 9-bar great staffs. The number of bars of each great staffafter layout change is uniquely determined by the number of sheet musicpages to be laid out into 1-page sheet music, the number of greatstaffs, and the number of bars of one great staff. It is, therefore,preferable to calculate in advance the number of bars of each greatstaff after layout change.

The interval between the treble and base staffs of each great staff isnot constant, and the width of a bar is not constant. Thus, it isdesirable to resize each block in layout so as to make the treble andbase staffs match each other.

Accordingly, even when laying out and printing a plurality of pages onone paper sheet, a printed material with good visibility can be created.

Second Embodiment

A second embodiment of the present invention will be described belowreferring to the drawings. In the first embodiment, the original imageis sheet music having a fixed format. In the second embodiment, theoriginal image is a document having no fixed format.

FIG. 14 is a flowchart showing an operation sequence in block numberingprocessing in step S240 of FIG. 6.

First, in step S310, grouping processing is executed to group blocksdivided in step S230 of FIG. 6. The grouping processing is for dividingan original image in the longitudinal direction at breaks between blocksin the longitudinal direction, thereby combining blocks into a pluralityof groups each including one or a plurality of blocks. A concreteexample of the grouping processing will be explained with reference toFIGS. 15 to 17. FIG. 15 shows the original image of one page. Referencenumerals 1501, 1502, 1503, 1504, and 1505 schematically denote texts;and 1506 and 1507, objects such as a graph, figure, and natural image.FIG. 16 is a view schematically showing numbered blocks obtained bydividing the original image of FIG. 15 into blocks by the methoddescribed in the first embodiment. FIG. 17 shows the result of groupingthe blocks of FIG. 16. The blocks are combined into 10 groups 1701 to1710.

Then, in step S320, group division processing is executed to divide agroup. The group division processing is to divide a group in the lateraldirection at a break between blocks in the lateral direction in a groupmade up of a plurality of blocks among groups. A concrete example of thegroup division processing will be explained with reference to FIG. 18.FIG. 18 shows an original image obtained by dividing, into groups, anoriginal image having undergone grouping as shown in FIG. 17. The group1705 in FIG. 17 is divided into groups 1711 and 1712. The group 1706 isdivided into groups 1713 and 1714. An example of laying out these groupsis shown in FIG. 22 or 23 to be described in the third embodiment.

In step S330, a block separated as the “text part” in step S230 of FIG.6 is divided based on the number of pages of the original image used inlayout printing. The text can be laid out in multiple columns bydividing a block separated as the “text part” into a plurality of blocksand rearranging them. In contrast, if a block separated as the “naturalimage part”, which is an image other than the text, is divided andrearranged, the image becomes meaningless. FIG. 19 shows an example ofdividing a block of the “text part” when laying out and printing twopages on one paper sheet. FIG. 19 shows an original image obtained bydividing a block of the text part in an original image having undergonegroup division processing as shown in FIG. 18. A block 1715 (1-1 left),a block 1716 (1-1 center), and a block 1717 (1-1 right) in FIG. 19 areobtained by dividing the group 1701 of the “text part” in FIG. 18 inaccordance with the number of pages to be laid out. In this case, theoriginal images of two pages are laid out and printed on one page.However, the block is divided into three to fit the page width whenchanging the scaling ratio of each block and arranging the block.

According to the same procedure, each of the groups 1702, 1703, 1704,1707, 1708, 1709, and 1710 of the “text part” in FIG. 18 is similarlydivided into three blocks. FIG. 25 shows the layout of these blocks. InFIG. 25, each of the groups 1702, 1703, 1704, 1707, 1708, 1709, and 1710of the “text part”, each of which is divided into three blocks, is laidout in three columns while maintaining the continuity of the text. Thislayout improves the readability of the “text part”, compared with thelayouts in FIGS. 22 and 23.

In this fashion, blocks are divided based on the result of image areaseparation. Even when laying out and printing an original image havingno fixed format, like a document created by a PC or a Web page, aprinted material with good visibility can be generated. The secondembodiment is directed to a case where neither information which becomesunnecessary in layout nor information which becomes necessary in layoutis detected. For example, when an original image contains a page number,the page number of the original image can be regarded as unnecessaryinformation to exclude the page number from the layout.

Third Embodiment

A third embodiment of the present invention will be described belowreferring to the drawings. FIG. 20 is a flowchart showing a processingsequence to display a layout result and determine a layout.

When laying out and printing the original images of a plurality of pageson one page according to the present invention, an optimal layoutdiffers depending on the user. For example, as for the text layout asshown in FIG. 21, a user may feel that a layout 2101 of one column ispreferable, but another user may feel that a layout 2102 of two columnsis preferable. The present invention does not simply rearrange aplurality of pages, so an infinite number of layouts are conceivable. Itis difficult to automatically provide a layout optimal for each userfrom an infinite number of layouts. Thus, a means for presenting aplurality of layouts to a user, and prompting him to select a layoutoptimal for him from these layouts is required.

First, in step S410, a count M which is determined in accordance with auser instruction and at which the layout result is displayed is set as aCount value.

Then, in step S420, layout change processing is done. The layout changeprocessing is, for example, layout change processing in step S290 ofFIG. 6. One layout is selected from a plurality of conceivable layoutsbased on a predetermined algorithm, and the current layout is changed tothe selected one. When the count at which the layout result is displayedis set to two or more in step S410, the predetermined algorithm is setto select a different layout every time the process shifts to this step.

In step S430, the layout result changed in step S420 is displayed.

In step S440, it is determined whether the layout result display counthas reached the count set by the user in step S410. If the layout resultdisplay count has reached this count (NO in step S440), the layoutresult is displayed, and the processing for determining a layout ends.If the layout result display count has not reached this count (YES instep S440), the process advances to step S450. When the layout result isdisplayed and the processing for determining a layout ends, conventionallayout printing of laying out and printing an original image for eachpage is executed.

In step S450, the Count value is decremented, and the process advancesto step S460.

In step S460, the user determines whether to employ the layout displayedin step S430. If the user determines via a UI to select the layoutresult (YES in step S460), the layout result is selected, and theprocess ends. If the user does not select the layout result (NO in stepS460), the process returns to step S420.

An example of a layout result displayed in step S430 of FIG. 20 whenlaying out and printing the original images of four pages on one papersheet will be explained with reference to FIGS. 22 and 23. A page layoutmethod is the same between the original images of four pages, so thelayout method for the first page will be described. Assume that theoriginal image of the first page is the original image in FIG. 15.

In FIG. 16 schematically showing numbered blocks, assume thatunnecessary information is discarded and necessary information isappended based on layout information of FIGS. 22 and 23.

FIG. 22 shows an example of a layout result displayed in step S430 ofFIG. 20. This layout result is obtained by laying out blocks based onthe layout rule in the second embodiment. FIG. 23 shows an example of alayout result different from that in FIG. 22, which is displayed becausethe user does not select the layout in FIG. 22. This layout result isobtained by laying out some blocks of the “text part” based on a layoutrule different from that in the second embodiment. The differencebetween FIGS. 22 and 23 is the difference in how to lay out blocks 1501to 1505. The blocks 1501 to 1505 are laid out in a different way basedon an image area separation result representing that all the blocks 1501to 1505 are determined as the “text part” in block conversion.

As described above, when dividing a plurality of pages into blocks andlaying out the blocks again, an infinite number of layouts areavailable. An optimal layout differs depending on the user. For thisreason, a plurality of layout rules are prepared. The user is presentedwith layout results processed based on these layout rules, and promptedto select a layout optimal for him. As a result, an optimal layoutresult can be provided.

Other Embodiments

The present invention is also carried out by supplying the program codesof software for implementing the functions of the above-describedembodiments to, for example, an apparatus connected to various devices,and operating these devices in accordance with programs stored in thecomputer of the apparatus or the like.

In this case, the program codes of the software implement the functionsof the above-described embodiments, and the program codes themselves andmeans such as a storage medium for supplying the program codes to acomputer constitute the present invention. Concrete examples of thestorage medium are a flexible disk, hard disk, optical disk,magnetooptical disk, CD-ROM, magnetic tape, nonvolatile memory card, andROM.

The present invention includes program codes when the functions of theabove-described embodiments are implemented by executing the programcodes supplied to the computer. The present invention also includesprogram codes when the functions of the above-described embodiments areimplemented by an OS (Operating System) and another application softwareor the like running on the computer in cooperation with each other.

The functions of the above-described embodiments are also implementedwhen program codes supplied to the computer are stored in the memoriesof a function expansion board and function expansion unit of thecomputer, and, for example, the CPUs of the function expansion board andfunction expansion unit execute processing based on the instructions ofthe program codes. The present invention also incorporates these programcodes.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2007-331071, filed on Dec. 21, 2007, which is hereby incorporated byreference herein in its entirety.

1. An image processing method of laying out and outputting an image ofan original formed from a plurality of pages on one page, the methodcomprising: inputting image data of the original formed from theplurality of pages; dividing the image data input in the inputting intoa plurality of blocks; selecting unnecessary information from each blockobtained by division in the dividing based on a predetermined rule tomaintain continuity of the original formed from the plurality of pageswhen laying out, on one page, the image data of the original formed fromthe plurality of pages; discarding the unnecessary information selectedin the selecting; appending necessary information to each block obtainedby division in the dividing based on the predetermined rule; replacingthe respective blocks obtained by division in the dividing with theblock from which the unnecessary information is discarded in thediscarding, and the block to which the necessary information is appendedin the appending; and outputting the respective blocks which haveundergone replacement in the replacing by laying out the respectiveblocks on one page.
 2. The method according to claim 1, wherein theoriginal includes sheet music, and the dividing divides an image of agreat staff of the sheet music into bars.
 3. The method according toclaim 2, wherein the unnecessary information includes at least one of apage number, a G clef, and an F clef.
 4. The method according to claim3, wherein the necessary information includes at least one of a G clefand an F clef.
 5. The method according to claim 1, wherein the originalincludes an image formed from a text and an image formed from an objectother than a text, and the dividing divides the image of the original sothat the image formed from the text and the image formed from the objectother than the text respectively belong to different blocks.
 6. Themethod according to claim 5, wherein the dividing divides the image ofthe original so that a continuous image formed from the object otherthan the text belongs to one block.
 7. The method according to claim 5,wherein the outputting includes: displaying an image in which therespective blocks which have undergone replacement in the replacing arelaid out on one page, and prompting a user to select whether or not toemploy the image, displayed in the displaying, in which the respectiveblocks are laid out on one page.
 8. An image processing apparatus whichlays out and outputs an image of an original formed from a plurality ofpages on one page, the apparatus comprising: an input unit which inputsimage data of the original formed from the plurality of pages; adivision unit which divides the image data input by the input unit intoa plurality of blocks; a selection unit which selects unnecessaryinformation from each block obtained by division by the division unitbased on a predetermined rule to maintain continuity of the originalformed from the plurality of pages when laying out, on one page, theimage data of the original formed from the plurality of pages; a discardunit which discards the unnecessary information selected by theselection unit; an appending unit which appends necessary information toeach block obtained by division by the division unit based on thepredetermined rule; a replacement unit which replaces the respectiveblocks obtained by division by the division unit with the block fromwhich the unnecessary information is discarded by the discard unit, andthe block to which the necessary information is appended by theappending unit; and an output unit which outputs the respective blockswhich have undergone replacement by the replacement unit by laying outthe respective blocks on one page.
 9. A computer-readable mediumcontaining computer-executable instructions utilized in an imageprocessing apparatus configured to lay out and output an image of anoriginal formed from a plurality of pages on one page, thecomputer-readable medium comprising: computer-executable instructionsfor inputting image data of the original formed from the plurality ofpages; computer-executable instructions for dividing the image datainput in the inputting into a plurality of blocks; computer-executableinstructions for selecting unnecessary information from each blockobtained by division in the dividing based on a predetermined rule tomaintain continuity of the original formed from the plurality of pageswhen laying out, on one page, the image data of the original formed fromthe plurality of pages; computer-executable instructions for discardingthe unnecessary information selected in the selecting;computer-executable instructions for appending necessary information toeach block obtained by division in the dividing based on thepredetermined rule; computer-executable instructions for replacing therespective blocks obtained by division in the dividing with the blockfrom which the unnecessary information is discarded in the discarding,and the block to which the necessary information is appended in theappending; and computer-executable instructions for outputting therespective blocks which have undergone replacement in the replacing bylaying out the respective blocks on one page.